(19) 




Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(11) EP 1 134 975 A2 

EUROPEAN PATENT APPLICATION 



(43) Date of publication: 

19.09.2001 Bulletin 2001/38 

(21) Application number: 01302289.2 

(22) Date of filing: 13.03.2001 



(51) lntCI.7: H04N 5/92 



(84) Designated Contracting States: 


(72) Inventors: 


AT BE CH CY DE DK ES Fi FR GB GR IE IT LI LU 


• Jun, Sung Bae 


MC NL PT SE TR 


Seoul (KR) 


Designated Extension States: 


• Yoon, Kyoung Ro 


AL LT LV MK RO SI 


Seoul (KR) 


(30) Priority: 13.03.2000 KR 2000012489 


(74) Representative: 


(71) Applicant: LG ELECTRONICS INC. 


McLelsh, Nicholas Alistair Maxwell et al 


Boult Wade Tennant 


Seoul (KR) 


Verulam Gardens 




70 Gray's Inn Road 




London WC1X 8BT (GB) 



(54) Non-linear reproduction control method of multimedia stream and apparatus tliereof 



(57) The present invention relates to a non-linear re- 
production control method of multimedia stream and an 
apparatus thereof, the present invention comprises 
judging a skip direction by interpreting whether a sl<ip 
request is a forward sl<ip or a backward skip when the 
skip request is inputted from the user, detennlning a skip 
unit by judging whether the user request is a big unit or 



a small unit, selecting a structural infomiation or seman- 
tic information change point of the multimedia stream 
mostly adjacent to the present reproduction position as 
a reproduction resume point on the basis of the deter- 
mined skip direction and skip unit or selecting the repro- 
duction resume point by using the set temporal offset 
fnam the preset reproduction position, accordingly it is 
possible to. 
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Description 

BACKGROUND OF THE INVENTION 

1 . Field of the Invention 

[0001] The present Invention relates to a method of 
controlling reproduction of a multimedia stream and an 
apparatus thereof, in particular to a non-linear reproduc- 
tion control method of a multimedia stream and an ap- 
paratus thereof including forward and a backward skip 
functions using structural information or semantic infor- 
mation of multimedia content. 

2. Description of the Prior Art 

[0002] Recently, according to development of a digital 
video technology and an image or a video processing 
technology, users can search and browse a request part 
of request multimedia content (drama, movie, documen- 
tary etc) at a request time point. 
[0003] A shot segmentation technology and a shot 
clustering technology are basic technologies for per- 
forming a non-linear multimedia browsing and search- 
ing. 

[0004] The shot segmentation is a technology for di- 
viding the video stream into each separated shot. 
[0005] The shot clustering technology is for construct- 
ing multimedia content by binding the each shot with log- 
ical scenes on the basis of timelimage/motion/audio/se- 
mantic similarity. 

[0006] The shot means a sequence of video frames 
taken from a camera without intenruption, it is the most 
basic unit for analyzing or constructing video, and the 
video is constructed with lots of shots. 
[0007] It Is possible to browse only Intersected section 
In the video content with the help of structural or seman- 
tic Index of a multimedia content. 
[0008] Instance replay function is the most frequently 
used in non-linear multimedia content browsing. 
[0009] When a user watches or listens a video or an 
audio program, the user can watch or listen again an 
interesting part or a part broadcast for absence by using 
the instant replay function, the instant replay function 
can be used In both analog and digital environment. 
[0010] Recently, It is possible to provide the Instant 
replay function notonly about video/audio recorded with 
an offline but also about a program broadcast or serv- 
iced in real-time by the development of a set-top box (Ex 
Tivo, replay TV) which is capable of performing record- 
ing and reproduction concurrently. 
[0011] Generally, methods for providing the instant re- 
play function uses a set temporal offset using average 
user reaction time. 

[0012] When the instant replay Is requested, in the 
method using the average user reaction time, reproduc- 
tion resumes from a time point which is about 7 ~ sec- 
onds previous to the present position by using the aver- 



age time (7 ~ 8 seconds) which is the average program 
miss time of the user in audition. In order to reproduce 
the video program from more previous scene, the user 
can reproduce the request part by performing the instant 
5 replay function a plurality of times. Because it is difficult 
to reproduce the exact starting position of the segment 
what the user want by using only the conventional in- 
stant replay function, a rewind function is usually used 
together. 

10 [001 3] In other words, In the conventional instant re- 
play using the average user reaction time, the reproduc- 
tion is performed by resume reproduction from the point 
which is fixed times before from the present reproduc- 
tion, accordingly the method can not be used when the 

'5 user requests the instant replay based on the structural 
or semantic information of a multimedia content. 
[0014] There is methods similar to the instant replay 
function using the preset temporal offset such as a 
method using temporal data blank, a method providing 

20 an instant replay function by designation of a user, and 
a method using physical storing position Information of 
a stream. 

[0015] The method using the temporal data blank is 
used in most of music programs, when a plurality of 
25 songs are recorded on a magnetic tape, transferring to 
the previous song or after song Is performed by search- 
ing a blank region of the temporal data by using a char- 
acteristic as a certain time Interval of data blank which 
exists between one song and the other song. 
30 [0016] However, the method using the temporal data 
blank can not be used for searching inner part of a 
stream in data having no temporal blank such as the 
multimedia stream, it can be used only for searching a 
limited unit stored In a limited media such as a magnetic 
35 tape for music. In addition, because its control unit Is 
very big, It Is difficult to use the method for a non-linear 
reproduction control function required by the user In the 
general video or audio program. In other words, the 
method using the temporal data blank is not the repro- 
40 duction control method for reproducing from the inner 
side of the video stream but the reproduction control 
method for reproducing by the stream unit. 
[0017] The instant replay function supply method by 
the user designation is a method designating a repre- 
ss duction start point before the user requests the instant 
replay, and Is mainly used for listening a certain part re- 
peatedly inside of a stream such as a tape for language 
study. 

[0018] Because the reproduction start point has to be 
50 designated in the method for providing the instant replay 
function by the user designation, it is very difficult to use 
the method for generic video or audio program. 
[0019] The method using the physical storing position 
information of the video stream is a method used for a 
55 digital media such as a disk for music, because the 
physical start position about each song is recorded on 
header information of the disk, it can react to a song unit 
navigation of the user. 
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[0020] The method using the physical storing position 
information of the video stream is capable of performing 
easily the searching by the video stream unit, however 
the Instant repiay control unit Is very big and it can not 
correspond to a reproduction control request by regions 
inside of the video stream, accordingly it is difficult to 
use the method for the reproduction control method 
about the generic video or audio program, and the meth- 
od only can provide a reproduction control function by 
the stream unit like the method using the temporal data 
blank. 



inputted by a user to a main control unit, a media file 
constructed with multimedia stream data, an index 
structure describing the structural information or seman- 
tic information of the pertinent multimedia content, and 
5 the main control unit for controlling the media file repro- 
duction in accordance with the order inputted from the 
input unit. 



BRIEF DESCRIPTION OF THE DRAWINGS 

10 

[0025] 



SUMMARY OF THE INVENTION 

[0021] It \Nou\A be desirable to provide a non-linear 
reproduction control method of a multimedia stream and 
an apparatus thereof w/hich is capable of skipping to a 
request point in a video/audio program by using struc- 
tural information or semantic Infbrmatlon of the multime- 
dia content. 

[0022] A non-linear reproduction control method of a 
multimedia stream in accordance with the present in- 
vention comprises judging a skip direction by Interpret- 
ing whether a skip request is a forward skip or a back- 
ward skip when the skip request is inputted from the us- 
er, determining a skip unit by judging whether the user 
request is a big unit or a small unit, selecting a structural 
information or semantic information alternation point of 
the multimedia stream mostly adjacent to the present 
reproduction position as a reproduction resuming point 
on the basis of the determined skip direction and skip 
unit, and resuming reproduction of the media from the 
selected point. 

[0023] A non-linear reproduction control method of a 
multimedia stream in accordance with another embod- 
iment of the present Invention comprises inputting a for- 
ware) or backward skip order, b, selecting a reproduction 
resuming point by considering the present reproduction 
position, skip direction, structural Information or seman- 
tic information of the multimedia stream or preset tem- 
poral offset information, reproducing the media from the 
selected reproduction resuming point, displaying addi- 
tional reproduction resuming candidate points using the 
structural or semantic information alternation point or 
point calculated by using a multiple of the temporal off- 
set. The reproduction resuming candidate polnt(s) will 
be display on a screen by using a key-frame or a key- 
region, reproduction will be performed when the user se- 
lects the key-frame or key-region displayed on the 
screen. 

[0024] A non-linear reproduction control apparatus of 
a multimedia stream in accordance with the Invention, 
for reproducing controlling by considering the present 
reproduction position, skip direction, structural Informa- 
tion or semantic information of the multimedia stream or 
preset temporal offset information, comprises a repro- 
duction unit for reproducing the multimedia stream, an 
input unit for transmitting a media file reproduction order 



FIG.1 Is a system block diagram for performing a 
non-linear reproduction control of a multimedia 

^5 stream in accordance with the present invention. 

FIG. 2 is a block diagram Illustrating structural in- 
formation of a video stream in accordance with an 
embodiment of the invention; 
FIG.3 Illustrates a semantic information index for a 

20 multimedia stream in accordance with an embodi- 
ment of the present Invention. 
FIG.4 Illustrates a video browsing Interface using an 
multimedia indexing technique for a multimedia 
stream, In accordance with an embodiment of the 

2s present invention. 

FIG. 5 illustrates an embodiment of an instant replay 
reproduction control method using structural infor- 
mation of content In accordance with the present 
Invention. 

30 FIG. 6 Illustrates an embodiment of an instant replay 
reproduction control method using semantic index 
information in accordance with the present inven- 
tion. 

FIG.7illustratesan embodiment of an Instant replay 
35 function using structural Information and semantic 
index information at the same time In accordance 
with the present invention. 
FIG. 8 Illustrates an Instant replay function using us- 
ing structural information or semantic information in 
40 accordance with the present Invention on the com- 
parison with the prior art. 

FIG.9A and 9B illustrate embodiments of a repro- 
duction control method in case of an instant replay 
function request in a news video. 
45 FIG.1 OA and 10B illustrate a screen construction 
method in an instant replay request In accordance 
with the present invention. 
FIG.11 illustrates a forward/backward skip position 
using structural information and semantic Informa- 
50 tion for a multimedia stream in accordance with the 
present invention. 

DETAILED DESCRIPTION OF THE PREFERRED 
EMBODIIVIENT 

55 

[0026] A non-linear reproduction control method of a 
multimedia stream and an method thereof will now de- 
scribed In detail with reference to accompanying draw- 
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ings. 

Fig. 1 is a system bloci< diagram for performing a non- 
linear reproduction control of a multimedia stream In ac- 
cordance with the present invention. 
[0027] It comprises an input unit 200 for transmitting 
an input by a user to a control module 400 in order to 
reproduce a media file 300, a control unit 400 for con- 
trolling a reproduction unit 100 and the media file 300 
by a signal outputted from the input unit 200, the repro- 
duction unit 1 00 for reproducing a multimedia stream by 
the control of the main control unit 400, the media file 
300 having multimedia stream data, and an Index struc- 
ture 500 describing structural information or semantic 
information about the multimedia content. 
[0028] The reproduction unit 10Q is a device used for 
reproducing the multimedia stream which can be a dis- 
play device, an audio device (Ex speal<er) etc. as the 
reproduction unit 100. 

[0029] The main control unit 400 comprises a control 
module 410 and a non-volatile memory 420. 
[0030] A software program controlling the playback of 
the media file 300 can be stored in the non-volatile mem- 
ory 420, content about a basic forward, rewind speed, 
unit time of a forward/backward skip and the number of 
boundary points to jump by a big unit skipping and a 
small unit in the forward/backward skipping etc. can be 
reconded in the non-volatile memory 420. 
[0031] The operation of the non-linear reproduction 
control apparatus of the multimedia stream will now be 
described as below. 

[0032] The user inputs a big unit/small unit forward/ 
backward skip order by using methods such as a button 
operation/mouse operation/voice input/gesture input 
etc. When the order is inputted to the control module 
410, the control module 410 calculates a resuming po- 
sition by considering the present position, a skip direc- 
tion and a skip unit and reads the structural information 
and semantic information stored in Index structure 500. 
After a reproduction point (file pointer etc.) of the media 
file 300 is adjusted, media file 300 reproduction to the 
reproduction unit 1 00 by reproducing the media from the 
resuming point. 

[0033] In ordering the forward/backward reproduction 
by using the button of the input unit such as a remote- 
controller etc., the user uses the each different button 
about the big unit skip and small unit skip, and the big 
unit skip and small unit skip can be divided by setting 
different operations on the same button. 
[0034] in setting the different operations on the same 
button, a method for recognizing the request operation. 
We can implement the forarard/backward skip with big 
unit and small unit with 2 buttons ; double clicks of the 
button Is interpreted as the big unit skip and one click of 
the button Is interpreted as the small unit skip or a short 
click of the button Is interpreted as the small skip and a 
long click of the button Is interpreted as the big unit skip 
can be used, it is possible also to use the opposite meth- 
od. 



[0035] The construction of the video stream for skip- 
ping by the big unit and small unit will now be described 
in detail with reference to accompanying drawings. 
[0036] FIG.2 Is a block diagram illustrating structural 
5 information of a video stream in accordance with the 
present invention. 

[0037] In other words, the video stream is divided into 
a plurality of scenes, the each scene is constructed with 
a plurality of sub-scenes or shots, the sub-scene has 

10 the characteristic of the scene. 

[0038] The shot Is a sequence of consecutive audio/ 
video frames, it means an editing unit, and the scene 
means a logical construction unit in the construction of 
the multimedia stream. 

15 [0039] The structural information of the multimedia 
stream is described with the shot and scene as the basic 
construction units, and temporal infomiation, namely, a 
start point and an end point or a start point and a length 
are described essentially in the each shot and scene. 

20 [0040] The indexing system Indexes the structural in- 
formation of the multimedia stream by extracting the 
shot, and detecting the scenes by using the extracted 
shot and additional information, 
[0041] The indexing system also indexes the seman- 

25 tic information exists In the multimedia stream. 

[0042] The semantic information about the multime- 
dia stream describes an appearance and disappear- 
ance of an object, an event occurrence, a state of an 
object, Information about a background etc. with time 

30 information. 

[0043] FIG. 3 illustrates an embodiment of a semantic 
infomiation indexing about a multimedia stream in ac- 
cordance with the present Invention. An Object, exists 
in ISq. S,], [S2, S4], [Sg, -]. 

35 [0044] The construction methods of the indexing 
structure will now be described with reference to accom- 
panying FIG.4 ~ 7. 

[0045] FIG.4 illustrates an embodiment of a video 
browsing interface using an index technique about a 
40 multimedia stream In accordance with the present in- 
vention, the user can access easily to the part of interest 
in the multimedia stream by using an interface having a 
TOG (Table of Content) form. 

[0046] FIG 5 illustrates an embodiment of an instant 
45 replay reproduction control method using structural in- 
formation of content in accordance with the present in- 
vention, the present reproduction position is a certain 
point of inside of a jth shot (Shotj.j) of a ith scene (Shotj.j) 
of the multimedia content. 
so [0047] Herein, when the user request the Instant re- 
play by the shot unit, the reproduction Is resumed from 
a 'c' point or a 'd' point, when the user request the Instant 
replay by the scene unit, the reproduction is resumed 
from a 'a' point or a 'b' point 
55 [0048] Herein, selecting a resuming point between 
the 'c' and 'd' or 'a' and 'b' can be determined by a meth- 
od designated by the system, and it Is also possible to 
provide resuming from the 'b' and 'd' points first and pro- 
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vide resuming from the 'a' or 'c' points when the instant 
replay is requested again within a certain time, 
[0049] Besides the instant replay control method us- 
ing the structural information of the multimedia stream, 
the instant replay function using the semantic informa- 
tion about the multimedia stream can be provided. 
[0050] The semantic information alternation point will 
now be described with reference to accompanying FIG. 
6. 

[0051] FIG.6 illustrates an embodiment of an instant 
replay reproduction control method using semantic in- 
dexing Information In accordance with the present in- 
vention In the most multimedia streams, alternation of 
the semantic Information occurs, for example, the object 
appears or disappears In the one scene/shot or through 
the plurality of shots, or an event occurs and ends etc. 
[0052] So ~ Sy in FIG.6 describe the semantic infor- 
mation alternation points, when the non-linear stream 
reproduction control method using the semantic Infor- 
mation Is used, the user can skip easily a uninteresting 
part and can reproduce easily fi-om a the starting point 
of segment of Interest by using the semantic index in- 
formation. 

[0053] In the instant replay function using the seman- 
tic Indexing information, when the instant replay by the 
small unit is requested, the reproduction is resumed 
from a semantic information alternation point which Is 
one point or two points previous to the present position, 
when the instant replay by the big unit Is requested, the 
alternation point of the semantic Information apart from 
the current reproduction position with predefined 
number of semantic information changes for big unit or 
an event change point or a background change point 
can be selected as a resuming position. 
[0054] In other words, when the instant replay by the 
small unit Is requested, the reproduction Is restarted 
form the present reproduction point to the Sg point, and 
when the Instant replay by the big unit is requested, the 
reproduction Is restarted from the Sg point where the 
event Is changed or S4 point where the setting Is 
changed. 

[0055] It is possible to apply the method using the skip 
unit by the structural information and method using the 
skip unit by the semantic infonnation at the same time. 
[0056] FiG.7 illustrates an embodiment of an instant 
replay function using structural information and seman- 
tic Information at the same time in accordance with the 
present Invention. 

[0057] When the Instant replay by the big unit is re- 
quested, a scene unit jump occurs, when the Instant re- 
play by the small unit is requested, the semantic infor- 
mation alternation point or a shot unit jump occurs. 
[0058] In other words, when the instant replay by the 
big unit is requested from the present reproduction point 
Shot|.j, the reproduction Is restarted from the 'a' or 'b'. 
When the instant replay by the small unit is requested, 
the reproduction is restarted from the 'c' or 'd' point di- 
vided by the structural Infonnation or from 'h' point di- 



vided by the semantic information. 
[0059] FIG. 8 illustrates an instant replay function us- 
ing structural infonnation or semantic information In ac- 
cordance with the present invention on the comparison 

5 with the prior art. 

[0060] In other words, most users use the instant re- 
play function when the reproduction restart to a 'z' or a 
y or a 'x' point is requested. 
[0061] In the instant replay request, In the conventlon- 

10 al technology, the Instant replay function is used several 
times In order to move the current reproduction position 
to the x, y, z points as described with an-ows. 
[0062] Accordingly, in order to resume the playback 
from start position of the segment of interest, the user 

15 has to use the instant replay function several times or 
use the other control function such as a rewind function. 
[0063] However, the control method in accordance 
with the present invention can restart the reproduction 
from the x, y, z points by moving current reproduction 

20 position to the x, y, z points directly by using structural 
Information or semantic information of a multimedia 
stream. 

[0064] The non-linear reproduction control method 
using the boundary point of the structural Information or 

25 semantic Information in accordance with the present In- 
vention can be used by being combining with the meth- 
od using the preset temporal offset In accordance with 
the conventional technology . 
[0065] The first combining method is selecting the re- 

30 production resuming point by considering the preset 
temporal offset with the skip direction, skip unit, the re- 
production resuming point is seleted as the point of the 
structural information or semantic information chagne 
point which Is mostly adjacent to a point calculated by 

35 using the preset temporal offset and the present repro- 
duction point. 

The second combining method Is selecting the repro- 
duction resuming point as the point calculated by using 
the preset temporal offset and the present reproduction 

40 point if the distance between the most adjacent strutural 
or semnatic information change point and the point cal- 
culated by using the preset temporal offset and the 
present reproduction point Is larger thant certain thresh- 
old, Otherwise sleeting the reproduction resuming point 

4S as the strutural or semnatic Information change point 
which Is the most adjacent to the point calculated by us- 
ing the preset temporal offset and the present reproduc- 
tion point. 

[0066] The examples of the instant replay function 
60 embodiment method using the structural Information or 
semantic information will now be described with refer- 
ence to accompanying FIG.9A and 9B. 
[0067] FIG.9A and 9B Illustrate embodiments of a re- 
production control method in case of an instant replay 
55 function requested during a news video browsing. FIG. 
9A describes about the reproduction resuming position 
when the instant replay is requested during browsing of 
a headline of a news in accordance with the present in- 
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vention and FIG.9B describes about the reproduction 
resuming position when the instant replay is requested 
during a news item browsing in accordance with the 
present invention. 

[0063] A thick boundary line describes a structural In- 5 
formation or semantic information change point in big 
unit, and a thin boundary line describes a structural In- 
formation or semantic information change point in small 
unit. 

[0069] The user can transfer to S-i or S2 or first posi- 10 
tlon of the head line Item 1^ or the start point of the head 
line news section I2 directly by requesting the Instant re- 
play function once or twice while browsing the head line 
section. 

[0070] A user may issue Instant replay in order to 15 
move the cun-ent reproduction position to the start point 
of the episode scene or to the start point of the news 
artilce In the Fig9B, user can directly move the current 
reproduction position to the start point of the current ep- 
isode shot 0/ or start point of the previous episode shot 20 
eg, or start point of the episode scene ei, or start point 
of new article ei directly. 

[0071] FIG.10A and 10B illustrate a screen construc- 
tion method when an instant replay requested in accord- 
ance with the present invention. FIG. 1 0A is a example 25 
video sequence for illustrating display interface related 
to the related to current Invention. Suppose that the cur- 
fent reproduction position is 'a' 
[0072] FiG.lOB describes the picture composition 
method in order to make the user select other resuming so 
position when he or she thlnl< the current resuming po- 
sition is not correct. Because there can be difference 
between the position the user actually requested and 
system recommended, system provides more candi- 
date resuming positions which is represented by key re- 35 
gion or key frame In order to enable a user to move the 
current reproduction position to what they actually want. 
If a user select a key frame or key region, the system 
will move the cun-ent reproduction position to the start 
poistlon of the sgement by which the key region or key -to 
frame represent. The each key-frame or key-region is 
In Input standby state, the one can be selected by a de- 
fault. 

[0073] In addition, an interi'ace which displays sepa- 
rately a big unit candidate point and a small unit candl- 45 
date point by dividing the skip unit into the big unit and 
small unit and differentiating vertical or horizontal ar- 
rangement of them can be adapted to the each key- 
frame or key-region. 

[0074] A third picture in FIG.10B uses a method dis- 50 
playing the key-frame or key-region about the candidate 
point vertically or horizontally by adjusting a size of a 
picture to be displayed, herein the key-frame/key-reglon 
Is In the Input standby state, the one can be selected by 
default. 55 
[0075] In addition, the candidate point in the big unit 
and candidate point in the small unit are provided to the 
user by being separated to vertically/horizontally or 



through additional methods In order to make the user 
know the point. 

[0076] The described picture composition method 
can be adapted to the fora/ard skip function also as well 
as the Instant replay (the backward skip). 
[0077] The non-linear reproduction control method 
using the structural information or semantic infonnatlon 
of the multimedia content in accordance with the present 
Invention can be used for forward skip as well as back- 
ward sklp(lnstant replay) In the method, a skip unit is 
determined by not only using the average user reaction 
time in the prior art, but also considering structural or 
semantic index at the same time, and a point where the 
structural information or semantic information alters is 
selected as a reproduction resuming position after skip. 
[0078] In addition, the fonvard/backward skip method 
using the structural information or semantic Information 
about the multimedia content in accordance with the 
present Invention can be arranged with the conventional 
method on the basis of the time. 
[0079] FIG. 1 1 illustrates a forward/backward skip po- 
sition using structural information and semantic Informa- 
tion about a multimedia stream in accordance with the 
present invention. 

[0080] A reproduction control method using structural 
information or semantic information about — multime- 
dia content in accordance with the present Invention can 
be adapted to content recorded with the offline, and can 
be applied to multimedia content broadcast or serviced 
In real-time. 

[0081] A reproduction control method using — struc- 
tural Information or semantic information about — mul- 
timedia content in accordance with the present invention 
can be adapted to all audio/video genres, Is possible to 
perform the reproduction control of the non-linear mul- 
timedia stream by the big unit and small unit, and can 
be applied to a skip of commercial advertisement. 
[0082] In addition, In order to perform the non-linear 
multimedia reproduction control, the present invention 
for providing the forward/backward skip can provide 
more convenient forward/backward skip function by pro- 
viding the interface using the key-frame or key-region 
when the user requests the forward/backward skip. 
[0083] Meanwhile, there is a plurality of Input means 
for requesting the Instant replay such as a button, a 
mouse, voice input, gesture input etc. can be used. As 
described above, embodiments of the invention enable 
users to skip a uninteresting part, browsing only inter- 
esting part. The present Invention can be implemented 
with very simple input device and a user can watch/listen 
the video In the optimum state. 



Claims 

1. A non-linear reproduction control method of a mul- 
timedia stream : comprising: 



6 



11 



EP1 134 975 A2 



12 



judging a skip direction by interpreting whether 
a sicip request is a forward skip or a backward 
skip when the skip request is inputted from the 
user ; 

determining a skip unit by judging whether the 
user request is a big unit or a small unit ; 
selecting a structural information or semantic 
information alternation point of the multimedia 
stream mostly adjacent to the present repro- 
duction position as a reproduction resuming 
point on the basis of the determined skip direc- 
tion and skip unit ; and 

resuming reproduction of the media from the 
selected point. 

2. The non-linear reproduction control method of the 
multimedia stream according to claim 1, wherein 
the structural Information in the selecting process 
for selecting the reproduction resuming point de- 
scribes Information about a shot or a scene existed 
in the multimedia stream on the basis of a time axis. 

3. The non-linear reproduction control method of the 
multimedia stream according to claim 1. wherein 
the semantic information In the selecting process 
for selecting the reproduction resuming point de- 
scribes information about appearance or disap- 
pearance of an object, state of the object, occur- 
rence or termination of an event, and background 
chagne information on the basis of the time axis. 

4. The non-linear reproduction control method of the 
multimedia stream according to claim 1, wherein 
the structural information or semantic infonnatlon 
about the multimedia stream in the selecting proc- 
ess for selecting the reproduction resuming point is 
video Index which Is extracted automatically by a 
video indexing system, described by semi-automat- 
ic or manual process. 

5. The non-linear reproduction control method of the 
multimedia stream according to claim 1, wherein 
the selecting process for selecting the reproduction 
resuming point by using the structural infonnatlon 
or semantic information about the multimedia 
stream comprises the step of: 

selecting a reproduction resuming point by 
considering the preset temporal offset with the skip 
direction, skip unit, the present reproduction point 
Information. 

6. The non-linear reproduction control method of the 
multimedia stream according to claim 5, wherein 
the selecting step for selecting the reproduction re- 
suming point by considering the preset temporal off- 
set with the skip direction, skip unit, the reproduc- 
tion resuming point is seleted as the point of the 
structural Information or semantic information 



change point which is mostly adjacent to a point cal- 
culated by using the preset temporal offset and the 
present reproduction point. 

5 7. The non-linear reproduction control method of the 
multimedia stream according to claim 5, wherein 
the selecting step for selecting the reproduction re- 
start point by considering the set temporal offset 
with the structural information or semantic informa- 

10 tlon, skip direction, skip unit, comprising: If distance 
between the most adjacent structural or semnatic 
Information change point and the point calculated 
by using the preset temporal offset and the present 
reproduction point is larger thant certain threshold, 

15 the reproduction resuming point is selected as a the 
point calculated by using the preset temporal offset. 
Otherwise, the reproduction resume point is select- 
ed as a the point is seleted as the strutural or se- 
mantic information change point which Is the most 

20 adjacent to the point calculated by using the preset 
temporal offset and the present reproduction point. 

8. The non-linear reproduction control method of the 
multimedia stream according to claim 1, wherein 

2S the selecting process for selecting the reproduction 
resuming point selects a shot unit as the skip unit 
In use of the structural Information, and selects an 
alternation point of the semantic information mostly 
adjacent to the present position as a transfeming 

30 point in use of the semantic information when a skip 
by the small unit is requested. 

9. The non-linear reproduction control method of the 
multimedia stream according to claim 1 , wherein 

35 the selecting process for selecting the reproduction 
resuming point selects a scene unit as the skip unit 
In use of the structural information, and selects an 
. alternation point of the semantic Information apart 
from the current reproduction position with prede- 

40 fined number of semantic information changes for 
big unit as a resuming point or an event change 
point or a background change point as a resuming 
position in use of the semantic information when a 
skip by the big unit Is requested. 

45 

10. A non-linear reproduction controf method of a mul- 
timedia stream : comprising : 

inputting a foro/ard or backward skip order from 
50 a user; 

selecting a reproduction resuming point by con- 
sidering the present reproduction position, a 
skip direction, structural infomiation or seman- 
tic information of the multimedia stream or pre- 
ss set temporal offset Information ; 

reproducing the media from the selected repro- 
duction resuming point; 
displaying additional reproduction resuming 
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candidate points using the structural or seman- 
tic information alternation point or point calcu- 
lated by using a multiple of the temporal offset 
besides the reproduction restart point on a 
screen by using a key-frame or a l<ey-region ; 
and 

resuming reproduction by considering the se- 
lected point as a reproduction restart point 
when the user selects the key-fl-ame or key-re- 
gion displayed on the screen. 

11. The non-linear reproduction control method of the 
multimedia stream according to claim 10, wherein 
the displaying process for displaying the additional 
reproduction resuming candidate points on the 
screen by using the key-frame or key-region dis- 
plays only the big unit or small unit on the screen 
between the skip units or displays the big unit and 
small unit on the screen at the same time, 

12. The non-linear reproduction control method of the 
multimedia stream according to claim 11, wherein 
a certain separator is displayed together on the 
screen in order to distinguish between the big unit 
and small unit when the big unit and small unit are 
displayed on the screen at the same time. 

13. The non-linear reproduction control method of the 
multimedia stream according to claim 11, wherein 
the big unit and small unit are displayed in temporal 
order without additional distinctions when the big 
unit and small unit are displayed on the screen at 
the same time. 

14. The non-linear reproduction control method of the 
multimedia stream according to claim 10, wherein 
the key-frame or key-region displayed on the 
screen In the displaying process is disappeared af- 
ter a certain time or by a request of the user. 

15. A non-linear reproduction control apparatus of a 
multimedia stream, comprising : 

a reproduction unit for reproducing the multime- 
dia stream ; 

an Input unit for transmitting a media file repro- 
duction order inputted by a user to a main con- 
trol unit, 

a media file constructed with multimedia stream 
data ; 

an index structure describing structural infor- 
mation or semantic information of the pertinent 
multimedia content ; and 
the main control unit for controlling the media 
file reproduction In accordance with the order 
inputted from the input unit. 

16. The non-linear reproduction control apparatus of 



the multimedia stream according to claim 15, 
wherein the input unit uses a method using a mouse 
or a button, a gesture Input method, and a voice or- 
der method etc. 

5 

17. The non-linear reproduction control apparatus of 
the multimedia stream according to claim 15, 
wherein the main controller comprises : 

10 a control module for controlling reproduction of 

the multimedia stream with the Input of the In- 
dex structure and media file, and controlling re- 
production of the whole multimedia stream by 
including a logic in order to perfonn judgement 

■'5 and control for reacting to a user request fl-om 

the input unit; and 

a non-volatile memory for storing an application 
program constructing a system, and storing fast 
forward/rewind speed for the user's request, 
20 preset time offset for forward or backward skip, 

and other user's preset parameter about media 
reproduction controls. 

18. The non-linear reproduction control apparatus of 
2s the multimedia stream according to claim 15, 

wherein the media file and Index structure is located 
In the local region of the system or connected to the 
system using a wire/wireless network 

30 19, The non-linear reproduction control apparatus of 
the multimedia stream according to claim 15, 
wherein different operations are permitted for the 
input unit In order to request the fonward or back- 
ward skip by the big unit/small unit by using the 

36 same button of the Input unit. 

20. The non-linear reproduction control apparatus of 

the multimedia stream according to claim 19, 
wherein the input unit recognizes the skip by the 

'fo small unit when the button is clicked once, or it rec- 
ognizes the skip by the big unit when the button is 
double clicked, or it recognizes the skip by the big 
unit when the button is clicked once, or it recognizes 
the skip by the small unit when the button is double 

45 clicked. 

21. The non-linear reproduction control apparatus of 
the multimedia stream accorting to claim 19, 
wherein the input unit recognizes the skip by the 

50 small unit when the button Is clicked shortly, or It 
recognizes the skip by the big unit when the button 
Is clicked long, or It recognizes the skip by the big 
unit when the button Is clicked shortly, or It recog- 
nizes the skip by the small unit when the button is 

55 clicked long. 
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